The GovLens crawler
If you run a website and saw GovLens-ingestion in your logs, this page is for you.
Who we are
GovLens (govlens.eu) is an EU legislative transparency service run by Trelvio Technology OÜ, Estonia. We follow EU and national legislation, parliamentary votes, lobbying registers and public spending, and link every figure to the official source it came from.
How we identify ourselves
Our automated requests send this user agent:
GovLens-ingestion/1.0 (+https://govlens.eu/bot)Some sites only accept agents in the long "compatible" form; for those we send Mozilla/5.0 (compatible; GovLensBot/1.0; +https://govlens.eu/bot). Both link to this page.
What we fetch
Only pages and files that are published openly by public bodies: EU institutions (Parliament, Council, Commission, EUR-Lex), national parliaments and gazettes, and official lobbying and spending registers. We do not log in, fill in forms that need an account, or fetch private content.
How often and how fast
Each source is fetched on a schedule, from once a day to a few times a day, and only when its data can have changed. We never open more than four simultaneous connections to one host, and most sources use one. Failed requests are retried a few times with increasing waits, then left until the next scheduled run.
Ask us to slow down or stop
Tell us and we will lower our rate, change the times we visit, or stop fetching a path or the whole site. If you would rather we use a feed, API or bulk file, tell us that too. Write to [email protected] with the host name, and what you would like us to do.
If your site blocks automated agents, please ask us to stop or write to us for access. Be aware that when a site blocks us, a fallback proxy may retry the request with a browser user agent.