Description:

I like news. I receive variety of news everyday. All news fall into email and sorted to different folders. Few years ago many news sites began to disable RSS support and now I have to open full version of sites to read the news. But these sites also posted their news inside Twitter, Facebook, Telegram. Because of this I wrote a tool which allow me grab news from these new data sources ... and fill my Elasticsearch with a new data too.

Quick start:


Clone and install:

git clone https://github.com/livelace/mosquito.git
cd mosquito
apt-get install phantomjs
pip install -r requirements.txt
pip install .

Save the configuration file (~/.mosquito.ini):

[main]

grab_timeout = 60

smtp_server = mail.example.com
smtp_port = 25
smtp_usessl = true
smtp_auth = true
smtp_from = mosquito@example.com
smtp_username = user@example.com
smtp_password = Passw0rD

verbose = info

[twitter]

consumer_key = <CONSUMER_KEY>
consumer_secret = <CONSUMER_SECRET>
access_token_key = <ACCESS_KEY>
access_token_secret = <ACCESS_SECRET>

List virtual machines:

mosquito list

+----+---------+---------+--------+---------------------+-----------------+-------------+--------+--------------------------+---------------------+
| ID | Enabled | Plugin  | Source | Destination         | Update interval | Description | Regexp | Regexp action            | Last update         |
+----+---------+---------+--------+---------------------+-----------------+-------------+--------+--------------------------+---------------------+
| 1  |   True  | twitter | rianru | o.popov@livelace.ru |       1s        | grab        |   .*   | grab=full                | 2016-09-16 12:27:37 |
|    |         |         |        |                     |                 |             |        | subject=Twitter          |                     |
|    |         |         |        |                     |                 |             |        | priority=high            |                     |
|    |         |         |        |                     |                 |             |        | header=X-mosquito:rianru |                     |
+----+---------+---------+--------+---------------------+-----------------+-------------+--------+--------------------------+---------------------+