90 lines
No EOL
3.5 KiB
Markdown
90 lines
No EOL
3.5 KiB
Markdown
# SMW Graph
|
|
|
|
This repository contains a script to pull semantic data out of Semantic Mediawiki and into a json. In turn, this json file can be loaded by graph.js to build a d3 graph.
|
|
|
|
Built for [Security Vision](https://securityvision.io).
|
|
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
pip install -r requirements.txt
|
|
wget https://d3js.org/d3.v6.min.js
|
|
```
|
|
|
|
## Update:
|
|
|
|
```
|
|
python wiki_relations.py
|
|
```
|
|
|
|
## Data
|
|
|
|
_Ask_ SMW with the following query:
|
|
|
|
```
|
|
({{#ask: [[Category:Deployments||Institution]]
|
|
OR [[Category:Technologies]] [[Developed by (institutions)::+]] // TODO: + should give a subquery <q></q> with all institutions in EU
|
|
OR [[Category:Technologies]] [[-Software Deployed::+]] // TODO: + should give a subquery <q></q> with all deployments in EU
|
|
|?Category
|
|
|?Geolocation
|
|
|?City
|
|
|?City.Has Coordinates=Has Coordinates
|
|
|?City.Is in Country=City Country
|
|
|?City Country.Has Coordinates=Country Coordinates
|
|
|?Clients
|
|
|?Managed by
|
|
|?Used by
|
|
|?Funded by
|
|
|?Provided by // TODO: technology/product provided by
|
|
|?Software Deployed
|
|
|?Developped by (institutions)
|
|
|format=broadtable
|
|
|limit=500
|
|
|offset=0
|
|
|link=all
|
|
|sort=
|
|
|order=asc
|
|
|headers=show
|
|
|searchlabel=... further results
|
|
|class=sortable wikitable smwtable
|
|
}}
|
|
|
|
TODO: see: https://www.semantic-mediawiki.org/wiki/Help:Querying_for_the_absence_of_a_property
|
|
|
|
{{#ask: [[Category:Deployments||Institution]] [[Is in Greens Report 2021::True]]
|
|
|?Category
|
|
|?Geolocation
|
|
|?City
|
|
|?City.Has Coordinates=City Coordinates
|
|
|?City.Is in Country=City Country
|
|
|?City Country.Has Coordinates=Country Coordinates
|
|
|?Institution Type
|
|
|?Clients
|
|
|?Managed by
|
|
|?Used by
|
|
|?Funded by
|
|
|?Provided by
|
|
|?Software Deployed
|
|
|?Software Deployed.Developped by (institutions)=Software Developer
|
|
|?Datasets Used
|
|
|?Datasets Used.Developed by Institution=Dataset Developer
|
|
|?Related Institutions
|
|
|?Involved Entities
|
|
|format=broadtable
|
|
|limit=50
|
|
|offset=0
|
|
|link=all
|
|
|sort=
|
|
|order=asc
|
|
|headers=show
|
|
|searchlabel=... further results
|
|
|class=sortable wikitable smwtable
|
|
}}
|
|
```
|
|
|
|
Fetch using CURL:
|
|
|
|
```bash
|
|
curl 'https://www.securityvision.io/wiki/index.php?title=Special:Ask&#search' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Content-Type: application/x-www-form-urlencoded' -H 'Origin: https://www.securityvision.io' -H 'Connection: keep-alive' -H 'Referer: https://www.securityvision.io/wiki/index.php/Special:Ask' -H 'Upgrade-Insecure-Requests: 1' --data-raw 'title=Special%3AAsk&_action=submit&q=%5B%5BCategory%3ADeployments%7C%7CInstitution%5D%5D++&po=%0D%0A+%7C%3FCategory%0D%0A+%7C%3FGeolocation%0D%0A+%7C%3FCity%0D%0A+%7C%3FCity.Has+Coordinates%3DCity+Coordinates%0D%0A+%7C%3FCity.Is+in+Country%3DCity+Country%0D%0A+%7C%3FCity+Country.Has+Coordinates%3DCountry+Coordinates%0D%0A+%7C%3FClients%0D%0A+%7C%3FManaged+by%0D%0A+%7C%3FUsed+by%0D%0A+%7C%3FFunded+by%0D%0A+%7C%3FProvided+by%0D%0A+%7C%3FSoftware+Deployed%0D%0A+%7C%3FSoftware+Deployed.Developped+by+%28institutions%29%3DSoftware+Developer%0D%0A+%7C%3FDatasets+Used%0D%0A+%7C%3FDatasets+Used.Developed+by+Institution%3DDataset+Developer&eq=yes&p%5Bformat%5D=json&p%5Blimit%5D=500&p%5Boffset%5D=0&p%5Blink%5D=all&p%5Bheaders%5D=show&p%5Bmainlabel%5D=&p%5Bintro%5D=&p%5Boutro%5D=&p%5Bsearchlabel%5D=JSON&p%5Bdefault%5D=&p%5Btype%5D=full&p%5Bfilename%5D=result2.json&sort_num%5B%5D=&order_num%5B%5D=asc&sort_num%5B%5D=&order_num%5B%5D=asc&eq=yes'
|
|
``` |