forked from security_vision/semantic_graph
initial pandoc export
This commit is contained in:
parent
815bb200be
commit
4bf9cfd954
2 changed files with 1955 additions and 44 deletions
49
README.md
49
README.md
|
@ -4,54 +4,21 @@ This repository contains a script to pull semantic data out of Semantic Mediawik
|
|||
|
||||
Built for [Security Vision](https://securityvision.io).
|
||||
|
||||
## Document
|
||||
|
||||
```
|
||||
pandoc EU\ Greens\ Report\ -\ Biometric\ Mass\ Surveillance\ v.3.docx --extract-media=images --template=template.html --section-divs -s -o output.html --metadata title="Biometric and Behavioural Mass Surveillance in EU Member States" -c mvp.css --toc --toc-depth=2
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
wget https://d3js.org/d3.v6.min.js
|
||||
```
|
||||
|
||||
## Update:
|
||||
|
||||
```
|
||||
python wiki_relations.py
|
||||
```
|
||||
|
||||
## Data
|
||||
|
||||
_Ask_ SMW with the following query:
|
||||
|
||||
```
|
||||
({{#ask: [[Category:Deployments||Institution]]
|
||||
OR [[Category:Technologies]] [[Developed by (institutions)::+]] // TODO: + should give a subquery <q></q> with all institutions in EU
|
||||
OR [[Category:Technologies]] [[-Software Deployed::+]] // TODO: + should give a subquery <q></q> with all deployments in EU
|
||||
|?Category
|
||||
|?Geolocation
|
||||
|?City
|
||||
|?City.Has Coordinates=Has Coordinates
|
||||
|?City.Is in Country=City Country
|
||||
|?City Country.Has Coordinates=Country Coordinates
|
||||
|?Clients
|
||||
|?Managed by
|
||||
|?Used by
|
||||
|?Funded by
|
||||
|?Provided by // TODO: technology/product provided by
|
||||
|?Software Deployed
|
||||
|?Developped by (institutions)
|
||||
|format=broadtable
|
||||
|limit=500
|
||||
|offset=0
|
||||
|link=all
|
||||
|sort=
|
||||
|order=asc
|
||||
|headers=show
|
||||
|searchlabel=... further results
|
||||
|class=sortable wikitable smwtable
|
||||
}}
|
||||
|
||||
TODO: see: https://www.semantic-mediawiki.org/wiki/Help:Querying_for_the_absence_of_a_property
|
||||
|
||||
{{#ask: [[Category:Deployments||Institution]] [[Is in Greens Report 2021::True]]
|
||||
|?Category
|
||||
|?Geolocation
|
||||
|
@ -81,10 +48,4 @@ TODO: see: https://www.semantic-mediawiki.org/wiki/Help:Querying_for_the_absence
|
|||
|searchlabel=... further results
|
||||
|class=sortable wikitable smwtable
|
||||
}}
|
||||
```
|
||||
|
||||
Fetch using CURL:
|
||||
|
||||
```bash
|
||||
curl 'https://www.securityvision.io/wiki/index.php?title=Special:Ask&#search' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Content-Type: application/x-www-form-urlencoded' -H 'Origin: https://www.securityvision.io' -H 'Connection: keep-alive' -H 'Referer: https://www.securityvision.io/wiki/index.php/Special:Ask' -H 'Upgrade-Insecure-Requests: 1' --data-raw 'title=Special%3AAsk&_action=submit&q=%5B%5BCategory%3ADeployments%7C%7CInstitution%5D%5D++&po=%0D%0A+%7C%3FCategory%0D%0A+%7C%3FGeolocation%0D%0A+%7C%3FCity%0D%0A+%7C%3FCity.Has+Coordinates%3DCity+Coordinates%0D%0A+%7C%3FCity.Is+in+Country%3DCity+Country%0D%0A+%7C%3FCity+Country.Has+Coordinates%3DCountry+Coordinates%0D%0A+%7C%3FClients%0D%0A+%7C%3FManaged+by%0D%0A+%7C%3FUsed+by%0D%0A+%7C%3FFunded+by%0D%0A+%7C%3FProvided+by%0D%0A+%7C%3FSoftware+Deployed%0D%0A+%7C%3FSoftware+Deployed.Developped+by+%28institutions%29%3DSoftware+Developer%0D%0A+%7C%3FDatasets+Used%0D%0A+%7C%3FDatasets+Used.Developed+by+Institution%3DDataset+Developer&eq=yes&p%5Bformat%5D=json&p%5Blimit%5D=500&p%5Boffset%5D=0&p%5Blink%5D=all&p%5Bheaders%5D=show&p%5Bmainlabel%5D=&p%5Bintro%5D=&p%5Boutro%5D=&p%5Bsearchlabel%5D=JSON&p%5Bdefault%5D=&p%5Btype%5D=full&p%5Bfilename%5D=result2.json&sort_num%5B%5D=&order_num%5B%5D=asc&sort_num%5B%5D=&order_num%5B%5D=asc&eq=yes'
|
||||
```
|
1950
doc/output.html
Normal file
1950
doc/output.html
Normal file
File diff suppressed because it is too large
Load diff
Loading…
Reference in a new issue