Spark in me - Internet, data science, math, deep learning, philosophy

@snakers4 Нравится 0

All this - lost like tears in rain.
Data science, ML, a bit of philosophy and math. No bs.
Our website
- http://spark-in.me
Our chat
- https://t.me/joinchat/Bv9tjkH9JHbxiV5hr91a0w
DS courses review
- http://goo.gl/5VGU5A
- https://goo.gl/YzVUKf
Гео и язык канала
Россия, Русский
Категория
Технологии


Написать автору
Гео канала
Россия
Язык канала
Русский
Категория
Технологии
Добавлен в индекс
09.05.2017 23:31
реклама
SearcheeBot
Ваш гид в мире Telegram-каналов
TGAlertsBot
Мониторинг упоминаний ключевых слов в каналах и чатах.
TGStat Bot
Бот для получения статистики каналов не выходя из Telegram
2 039
подписчиков
~2.1k
охват 1 публикации
~1.1k
дневной охват
~6
постов / нед.
105.3%
ERR %
4.37
индекс цитирования
Репосты и упоминания канала
9 упоминаний канала
30 упоминаний публикаций
50 репостов
Data Science by ODS.ai
Data Science by ODS.ai
Just links
Data Science by ODS.ai
Data Science by ODS.ai
Data Science by ODS.ai
Нейронач
Just links
Нейронач
Physics Blues
Нейронач
Just links
Just links
Just links
Админим с Буквой
Main ML_KZ
Нейронач
Нейронач
RE:post
Нейронач
Just links
Нейронач
Нейронач
Нейронач
Админим с Буквой
Just links
Anscombe's Quartet
Админим с Буквой
Админим с Буквой
Food-stained hoodie
Anscombe's Quartet
Блог Шмакова
Anscombe's Quartet
Anscombe's Quartet
Machinelearning
Отраженный свет
Dato ML
Dato ML
DeepLearning ru
Dato ML
Каналы, которые цитирует @snakers4
Silero API news
Silero API news
Silero API news
Silero API news
Silero API news
Silero API news
Silero API news
Silero API news
Silero API news
Матчасть
addmeto
Just links
Just links
DL in NLP
Just links
Just links
Just links
Just links
Neural Networks Engineering
Just links
Just links
Neural Networks Engineering
Neural Networks Engineering
Just links
Just links
Just links
Админим с Буквой
Вастрик.Пынь
Bird Born
Loss function porn
Just links
Just links
Админим с Буквой
Bird Born
NVIDIA
Админим с Буквой
Just links
Just links
Just links
Just links
Админим с Буквой
Hacker News
Just links
Админим с Буквой
NVIDIA
Just links
Hacker News
Just links
Админим с Буквой
Just links
Последние публикации
Удалённые
С упоминаниями
Репосты
Towards an ImageNet Moment for Speech-to-Text Part 2

Following our post on habr (https://habr.com/ru/post/494006/) and our first post dedicated to training practical STT models (https://thegradient.pub/towards-an-imagenet-moment-for-speech-to-text/) we are publishing a second post, more technical and dedicated to the STT industry and academia itself.

Please feel free to revisit these posts for more info / more info on our models:

- https://t.me/snakers4/2443
- https://t.me/snakers4/2431

Now, enjoy:

- A Speech-To-Text Practitioner’s Criticisms of Industry and Academia https://thegradient.pub/a-speech-to-text-practitioners-criticisms-of-industry-and-academia/

Please share and repost!

#speech
#deep_learning
Читать полностью
Towards End-to-end ASR

Towards End-to-end ASR - an internal (?) presentation by Google
https://drive.google.com/file/d/1Rpob1-C223L9UWTiLJ6_Dy12mTA3YyTn/view

This is such a huge corpus of work.

Interesting conclusions:

- Google writes your voice (at least from Google assistant, unclear whether they abuse their "phone" app) and uses this data for their models. Surprise surprise!

- Obviously Google is pushing towards end-to-end ASR within one NN on a mobile device for a number of reasons:

(i) easier packaging
(ii) quantization
(iii) no requirement to run a large LM alongside the model
(iv) Google has a lot of data (end-to-end models suffer from lack of data mostly)

- 120MB total system size on mobile device. This means AM + LM, which in this case is one quantized RNN-T model (4x - float32 => int8)

- They also write that hybrid systems with LM fusion / rescoring perform better overall

- The "best" cited solutions are not end-to-end, though

- Finally understood why they were pushing their RNN-T models instead of 10x more frugal alternatives. Old and optimized layers, hacks to speed up inference, unlimited resources, better performance (on the same step). Also LSTMs are known to be able to replace LMs

- Google also knows about "Time Reduction Layer", but looks like when using it within and RNN it is a bit painful - a lot of fiddling in the model logic

- Looks like given unlimited resources, data and compute - the easiest solution is to train large LSTMs in an end-to-end fashion (I also noticed that LSTMs have higher quality on same step, but MUCH weaker speed and convergence overall in terms of time-to-accuracy), optimize it heavily, quantize and deploy

- Sharing AMs / LMs for different dialects kind of works (maybe in terms of time-to-accuracy?), but direct tuning is better

But is full 100% end-to-end feasible on any scale below Google?

Probably not. Unless you are Facebook.
Having a fully end-2-end pipeline will have OOV (even with BPE / word-piece tokens) and other problems - like bias towards domains where you have audio. It will certainly NOT generalize towards unseen new words and pronunciations.
Meh.

But can you have extremely small mobile models?

Yes and no. Our latest small AM is targeting 200MB before quantization and probably 50MB after. Current production model is around 90MB (after quantization). But can it serve instead of an LM?

Technically yes, but quality will suffer. Unlike Google we do not have unlimited data, compute and low level engineers. On the other hand fully neural post-processing / decoding w/o huge transformer-like models is more than feasible. So we will see =)

#speech
Читать полностью
2020 DS / ML Digest 4

Highlights

- Google tries to ... reduce compute used for transformer pre-training
- Some NLP datasets - Russian news and Google dialogues
- Very cool story of a failed autonomous truck company
- The state of autonomous vehicles

Please like / share / repost!

https://spark-in.me/post/2020_ds_ml_digest_04

#digest
Towards an ImageNet Moment for Speech-to-Text

First CV, and then (arguably) NLP, have had their ImageNet moment ⁠— a technical shift that makes tackling many problems much easier. Could Speech-To-Text be next?

Following the release of our production models / metrics, this is our piece on this topic on thegradient.pub! So far this is the largest work ever we have done, and I hope that it will not go under the radar.

It is in our hands now to make sure that speech recognition brings value to people worldwide, and not only some fat cats.

So, without further ado:

- The piece itself https://thegradient.pub/towards-an-imagenet-moment-for-speech-to-text/
- Some more links here https://spark-in.me/post/towards-an-imagenet-moment-for-speech-to-text
- If you are on Twitter, please repost this message - https://twitter.com/gradientpub/status/1243967773635571712

A lot of thanks to Thegradient team, especially Andrey and Jacob, for the sheer amount of work they put in to make this piece readable and understandable!

Please like, share, repost!

Also, there will be a second piece with criticism, so stay tuned!

#speech
#deep_learning
Читать полностью
Surprisingly this did not work.

My guess - because of how CTC networks learn blanks and the signal is dilluted.

Just tuning the hyper-params gave me 40% further weight reduction, but the networks are still ~2pp CER off.

Maybe it will work with networks trained wo CTC?
Репост из: Silero API news
Репост из: Silero API news
Репост из: Silero API news
Репост из: Silero API news
Репост из: Silero API news
Репост из: Silero API news
Репост из: Silero API news
Репост из: Silero API news
Репост из: Silero API news
Also some pics, if you are too lazy to follow links
Russian Speech Recognition

You may have heard about our dataset, Open STT.

And yeah, if you have not guessed we have built a Speech-To-Text system that is better / on par with alleged "market leaders", the only difference being that we publish something for the common good and we do not need 100 Tesla GPUs (wink-wink, Oleg).

Also if it is not super obvious, this thing is already deployed into production and it really works.

Now we decided to go out of stealth mode a bit and publish a series of publications in online Russian / English publications:

- A piece on Habr.com - just published https://habr.com/ru/post/494006/ - it is very short and abridged, you know habr;
- 2 more detailed pieces on https://thegradient.pub - coming soon!

If you want more gory details, you can see a couple of posts on our project's website:

- STT system quality benchmarks - https://www.silero.ai/russian-stt-benchmarks/
- STT system speed https://www.silero.ai/stt-system-speed/
- How to measure quality in STT https://www.silero.ai/stt-quality-metrics/

If you would like to test our system, you may first want to:

- Try a demo http://demo.silero.ai/ (more beautiful mobile demo coming up!)
- See the API https://api.silero.ai/docs

#deep_learning
#speech
#nlp
Читать полностью
Which Notebooks Do You Use?
anonymous poll

Jupyter NOTEBOOK – 92
👍👍👍👍👍👍👍 63%

Jupyter LAB – 34
👍👍👍 23%

Some IDE plugin – 8
👍 5%

Jupyter HUB (with NOTEBOOKS) – 7
👍 5%

Your option – 4
▫️ 3%

Atom + hydrogen – 1
▫️ 1%

👥 146 people voted so far.
Jupyter NOTEBOOK – 63%
Jupyter LAB – 23%
Jupyter HUB (with NOTEBOOKS) – 5%
Some IDE plugin – 5%
Atom + hydrogen – 1%
Your option – 3%
Finally Migrating to JupyterLab?

TLDR - there is no killer feature, most likely this is just future-proofing.

With these plugins (some of which even work with the latest version of JupyterLab) you can finally migrate:

- jupyterlab_filetree
- toc
- collapsible_headings

Extensions can be installed using jupyter labextension install. Depending on your conda installation, sometimes you can even install them in your JupyterLab UI.

Migration from Notebook

Just add this and replace how you run your notebook:

RUN conda install -c conda-forge jupyterlab && \
conda install nodejs
...

CMD jupyter lab --port=8888 --ip=0.0.0.0 --no-browser

Obvious Downsides

- Looks like it is slower than notebooks (most annoying factor)
- No clear UX improvement, text editors are worse than IDEs, notebooks are the same
- Terminal is much less useful than standard Linux terminal or Putty
- With larger / more structured notebooks it crashes
- Most likely JupyterHub will continue to work with notebooks


I understand that pushing code to tested modules / having more smaller notebooks is preferable, but now when I have given this a test, most likely I will migrate only when forced to.


#data_science
Читать полностью
We tested this

Surprisingly, this was worse than just plain BPE on our task

=(