semantic_graph/README.md

# SMW Graph

This repository contains a script to pull semantic data out of Semantic Mediawiki and into a json. In turn, this json file can be loaded by graph.js to build a d3 graph.

Built for [Security Vision](https://securityvision.io).


## Installation

```bash
pip install -r requirements.txt
wget https://d3js.org/d3.v6.min.js
```

## Update:

```
python wiki_relations.py
```

## Data

_Ask_ SMW with the following query:

```
({{#ask: [[Category:Deployments||Institution]]  
OR [[Category:Technologies]] [[Developed by (institutions)::+]]  // TODO: + should give a subquery <q></q> with all institutions in EU
OR [[Category:Technologies]]  [[-Software Deployed::+]] // TODO: + should give a subquery <q></q> with all deployments in EU
 |?Category
 |?Geolocation
 |?City
 |?City.Has Coordinates=Has Coordinates
 |?City.Is in Country=City Country
 |?City Country.Has Coordinates=Country Coordinates
 |?Clients
 |?Managed by
 |?Used by
 |?Funded by
 |?Provided by // TODO: technology/product provided by
 |?Software Deployed
 |?Developped by (institutions)
 |format=broadtable
 |limit=500
 |offset=0
 |link=all
 |sort=
 |order=asc
 |headers=show
 |searchlabel=... further results
 |class=sortable wikitable smwtable
}}

TODO: see: https://www.semantic-mediawiki.org/wiki/Help:Querying_for_the_absence_of_a_property

{{#ask: [[Category:Deployments||Institution]]  [[Is in Greens Report 2021::True]]
 |?Category
 |?Geolocation
 |?City
 |?City.Has Coordinates=City Coordinates
 |?City.Is in Country=City Country
 |?City Country.Has Coordinates=Country Coordinates
 |?Institution Type
 |?Clients
 |?Managed by
 |?Used by
 |?Funded by
 |?Provided by
 |?Software Deployed
 |?Software Deployed.Developped by (institutions)=Software Developer
 |?Datasets Used
 |?Datasets Used.Developed by Institution=Dataset Developer
 |?Related Institutions
 |?Involved Entities
 |format=broadtable
 |limit=50
 |offset=0
 |link=all
 |sort=
 |order=asc
 |headers=show
 |searchlabel=... further results
 |class=sortable wikitable smwtable
}}
```

Fetch using CURL:

```bash
curl 'https://www.securityvision.io/wiki/index.php?title=Special:Ask&#search' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Content-Type: application/x-www-form-urlencoded' -H 'Origin: https://www.securityvision.io' -H 'Connection: keep-alive' -H 'Referer: https://www.securityvision.io/wiki/index.php/Special:Ask' -H 'Upgrade-Insecure-Requests: 1' --data-raw 'title=Special%3AAsk&_action=submit&q=%5B%5BCategory%3ADeployments%7C%7CInstitution%5D%5D++&po=%0D%0A+%7C%3FCategory%0D%0A+%7C%3FGeolocation%0D%0A+%7C%3FCity%0D%0A+%7C%3FCity.Has+Coordinates%3DCity+Coordinates%0D%0A+%7C%3FCity.Is+in+Country%3DCity+Country%0D%0A+%7C%3FCity+Country.Has+Coordinates%3DCountry+Coordinates%0D%0A+%7C%3FClients%0D%0A+%7C%3FManaged+by%0D%0A+%7C%3FUsed+by%0D%0A+%7C%3FFunded+by%0D%0A+%7C%3FProvided+by%0D%0A+%7C%3FSoftware+Deployed%0D%0A+%7C%3FSoftware+Deployed.Developped+by+%28institutions%29%3DSoftware+Developer%0D%0A+%7C%3FDatasets+Used%0D%0A+%7C%3FDatasets+Used.Developed+by+Institution%3DDataset+Developer&eq=yes&p%5Bformat%5D=json&p%5Blimit%5D=500&p%5Boffset%5D=0&p%5Blink%5D=all&p%5Bheaders%5D=show&p%5Bmainlabel%5D=&p%5Bintro%5D=&p%5Boutro%5D=&p%5Bsearchlabel%5D=JSON&p%5Bdefault%5D=&p%5Btype%5D=full&p%5Bfilename%5D=result2.json&sort_num%5B%5D=&order_num%5B%5D=asc&sort_num%5B%5D=&order_num%5B%5D=asc&eq=yes'
```