Graph from Semantic Mediawiki data
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
Ruben van de Ven 3c0184fe97 Show sources and fix arrow angle offset 10 months ago
www Show sources and fix arrow angle offset 10 months ago
.gitignore Basic graph version 1 year ago
README.md chores 1 year ago
graph.js simulation prerendering is now a config var 1 year ago
index.html WIP messy graph 1 year ago
no_credentials.json get rid of beginners mistake: credentials in external file 1 year ago
requirements.txt Basic graph version 1 year ago
wiki_relations.py Little ordering of graph 1 year ago

README.md

SMW Graph

This repository contains a script to pull semantic data out of Semantic Mediawiki and into a json. In turn, this json file can be loaded by graph.js to build a d3 graph.

Built for Security Vision.

Installation

pip install -r requirements.txt
wget https://d3js.org/d3.v6.min.js

Update:

python wiki_relations.py

Data

Ask SMW with the following query:

({{#ask: [[Category:Deployments||Institution]]  
OR [[Category:Technologies]] [[Developed by (institutions)::+]]  // TODO: + should give a subquery <q></q> with all institutions in EU
OR [[Category:Technologies]]  [[-Software Deployed::+]] // TODO: + should give a subquery <q></q> with all deployments in EU
 |?Category
 |?Geolocation
 |?City
 |?City.Has Coordinates=Has Coordinates
 |?City.Is in Country=City Country
 |?City Country.Has Coordinates=Country Coordinates
 |?Clients
 |?Managed by
 |?Used by
 |?Funded by
 |?Provided by // TODO: technology/product provided by
 |?Software Deployed
 |?Developped by (institutions)
 |format=broadtable
 |limit=500
 |offset=0
 |link=all
 |sort=
 |order=asc
 |headers=show
 |searchlabel=... further results
 |class=sortable wikitable smwtable
}}

TODO: see: https://www.semantic-mediawiki.org/wiki/Help:Querying_for_the_absence_of_a_property

{{#ask: [[Category:Deployments||Institution]]  [[Is in Greens Report 2021::True]]
 |?Category
 |?Geolocation
 |?City
 |?City.Has Coordinates=City Coordinates
 |?City.Is in Country=City Country
 |?City Country.Has Coordinates=Country Coordinates
 |?Institution Type
 |?Clients
 |?Managed by
 |?Used by
 |?Funded by
 |?Provided by
 |?Software Deployed
 |?Software Deployed.Developped by (institutions)=Software Developer
 |?Datasets Used
 |?Datasets Used.Developed by Institution=Dataset Developer
 |?Related Institutions
 |?Involved Entities
 |format=broadtable
 |limit=50
 |offset=0
 |link=all
 |sort=
 |order=asc
 |headers=show
 |searchlabel=... further results
 |class=sortable wikitable smwtable
}}

Fetch using CURL:

curl 'https://www.securityvision.io/wiki/index.php?title=Special:Ask&#search' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Content-Type: application/x-www-form-urlencoded' -H 'Origin: https://www.securityvision.io' -H 'Connection: keep-alive' -H 'Referer: https://www.securityvision.io/wiki/index.php/Special:Ask' -H 'Upgrade-Insecure-Requests: 1' --data-raw 'title=Special%3AAsk&_action=submit&q=%5B%5BCategory%3ADeployments%7C%7CInstitution%5D%5D++&po=%0D%0A+%7C%3FCategory%0D%0A+%7C%3FGeolocation%0D%0A+%7C%3FCity%0D%0A+%7C%3FCity.Has+Coordinates%3DCity+Coordinates%0D%0A+%7C%3FCity.Is+in+Country%3DCity+Country%0D%0A+%7C%3FCity+Country.Has+Coordinates%3DCountry+Coordinates%0D%0A+%7C%3FClients%0D%0A+%7C%3FManaged+by%0D%0A+%7C%3FUsed+by%0D%0A+%7C%3FFunded+by%0D%0A+%7C%3FProvided+by%0D%0A+%7C%3FSoftware+Deployed%0D%0A+%7C%3FSoftware+Deployed.Developped+by+%28institutions%29%3DSoftware+Developer%0D%0A+%7C%3FDatasets+Used%0D%0A+%7C%3FDatasets+Used.Developed+by+Institution%3DDataset+Developer&eq=yes&p%5Bformat%5D=json&p%5Blimit%5D=500&p%5Boffset%5D=0&p%5Blink%5D=all&p%5Bheaders%5D=show&p%5Bmainlabel%5D=&p%5Bintro%5D=&p%5Boutro%5D=&p%5Bsearchlabel%5D=JSON&p%5Bdefault%5D=&p%5Btype%5D=full&p%5Bfilename%5D=result2.json&sort_num%5B%5D=&order_num%5B%5D=asc&sort_num%5B%5D=&order_num%5B%5D=asc&eq=yes'