r/greens_report
1
0
Fork 0
forked from security_vision/semantic_graph
Code Issues Pull requests Projects Releases Wiki Activity

initial pandoc export

Browse source
This commit is contained in:
Ruben van de Ven 2021-10-07 08:43:56 +02:00
parent 815bb200be
commit 4bf9cfd954
2 changed files with 1955 additions and 44 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

49
README.md
View file

@ -4,54 +4,21 @@ This repository contains a script to pull semantic data out of Semantic Mediawik
Built for [Security Vision](https://securityvision.io).
## Document
```
pandoc EU\ Greens\ Report\ -\ Biometric\ Mass\ Surveillance\ v.3.docx --extract-media=images --template=template.html --section-divs -s -o output.html --metadata title="Biometric and Behavioural Mass Surveillance in EU Member States" -c mvp.css --toc --toc-depth=2
```
## Installation
```bash
pip install -r requirements.txt
wget https://d3js.org/d3.v6.min.js
```
## Update:
```
python wiki_relations.py
```
## Data
_Ask_ SMW with the following query:
```
({{#ask: [[Category:Deployments||Institution]]
OR [[Category:Technologies]] [[Developed by (institutions)::+]] // TODO: + should give a subquery <q></q> with all institutions in EU
OR [[Category:Technologies]] [[-Software Deployed::+]] // TODO: + should give a subquery <q></q> with all deployments in EU
|?Category
|?Geolocation
|?City
|?City.Has Coordinates=Has Coordinates
|?City.Is in Country=City Country
|?City Country.Has Coordinates=Country Coordinates
|?Clients
|?Managed by
|?Used by
|?Funded by
|?Provided by // TODO: technology/product provided by
|?Software Deployed
|?Developped by (institutions)
|format=broadtable
|limit=500
|offset=0
|link=all
|sort=
|order=asc
|headers=show
|searchlabel=... further results
|class=sortable wikitable smwtable
}}
TODO: see: https://www.semantic-mediawiki.org/wiki/Help:Querying_for_the_absence_of_a_property
{{#ask: [[Category:Deployments||Institution]] [[Is in Greens Report 2021::True]]
|?Category
|?Geolocation
@ -82,9 +49,3 @@ TODO: see: https://www.semantic-mediawiki.org/wiki/Help:Querying_for_the_absence
|class=sortable wikitable smwtable
}}
```
Fetch using CURL:
```bash
curl 'https://www.securityvision.io/wiki/index.php?title=Special:Ask&#search' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Content-Type: application/x-www-form-urlencoded' -H 'Origin: https://www.securityvision.io' -H 'Connection: keep-alive' -H 'Referer: https://www.securityvision.io/wiki/index.php/Special:Ask' -H 'Upgrade-Insecure-Requests: 1' --data-raw 'title=Special%3AAsk&_action=submit&q=%5B%5BCategory%3ADeployments%7C%7CInstitution%5D%5D++&po=%0D%0A+%7C%3FCategory%0D%0A+%7C%3FGeolocation%0D%0A+%7C%3FCity%0D%0A+%7C%3FCity.Has+Coordinates%3DCity+Coordinates%0D%0A+%7C%3FCity.Is+in+Country%3DCity+Country%0D%0A+%7C%3FCity+Country.Has+Coordinates%3DCountry+Coordinates%0D%0A+%7C%3FClients%0D%0A+%7C%3FManaged+by%0D%0A+%7C%3FUsed+by%0D%0A+%7C%3FFunded+by%0D%0A+%7C%3FProvided+by%0D%0A+%7C%3FSoftware+Deployed%0D%0A+%7C%3FSoftware+Deployed.Developped+by+%28institutions%29%3DSoftware+Developer%0D%0A+%7C%3FDatasets+Used%0D%0A+%7C%3FDatasets+Used.Developed+by+Institution%3DDataset+Developer&eq=yes&p%5Bformat%5D=json&p%5Blimit%5D=500&p%5Boffset%5D=0&p%5Blink%5D=all&p%5Bheaders%5D=show&p%5Bmainlabel%5D=&p%5Bintro%5D=&p%5Boutro%5D=&p%5Bsearchlabel%5D=JSON&p%5Bdefault%5D=&p%5Btype%5D=full&p%5Bfilename%5D=result2.json&sort_num%5B%5D=&order_num%5B%5D=asc&sort_num%5B%5D=&order_num%5B%5D=asc&eq=yes'
```

1950
doc/output.html Normal file
View file

File diff suppressed because it is too large Load diff