Skip to content
GovLens is in free beta — pick any plan, free while we build. Join the beta →
Crawler

The GovLens crawler

If you run a website and saw GovLens-ingestion in your logs, this page is for you.

Who we are

GovLens (govlens.eu) is an EU legislative transparency service run by Trelvio Technology OÜ, Estonia. We follow EU and national legislation, parliamentary votes, lobbying registers and public spending, and link every figure to the official source it came from.

How we identify ourselves

Our automated requests send this user agent:

GovLens-ingestion/1.0 (+https://govlens.eu/bot)

Some sites only accept agents in the long "compatible" form; for those we send Mozilla/5.0 (compatible; GovLensBot/1.0; +https://govlens.eu/bot). Both link to this page.

What we fetch

Only pages and files that are published openly by public bodies: EU institutions (Parliament, Council, Commission, EUR-Lex), national parliaments and gazettes, and official lobbying and spending registers. We do not log in, fill in forms that need an account, or fetch private content.

How often and how fast

Each source is fetched on a schedule, from once a day to a few times a day, and only when its data can have changed. We never open more than four simultaneous connections to one host, and most sources use one. Failed requests are retried a few times with increasing waits, then left until the next scheduled run.

Ask us to slow down or stop

Tell us and we will lower our rate, change the times we visit, or stop fetching a path or the whole site. If you would rather we use a feed, API or bulk file, tell us that too. Write to [email protected] with the host name, and what you would like us to do.

If your site blocks automated agents, please ask us to stop or write to us for access. Be aware that when a site blocks us, a fallback proxy may retry the request with a browser user agent.