Neural networks for log analysis?

Hi, Habr, at the University I went to work on the project, but very little information about it
The essence of the question - is it possible to create a system of trained neural networks for log analysis of the company in real time to identify any anomalous activity? and where can I read about it, because what I found is several short articles and videos. Thanks in advance for any information on the subject
April 4th 20 at 00:46
4 answers
April 4th 20 at 00:48
Solution
As you already said , "anyone" cannot. Certain types - it is possible. And, by the way, not only the logs for these conditions. Only here in one post on the forum about it will tell.
There are at least two overlap area of expertise is machine learning (and neural networks, as one of the tools used in it) and cybersecurity. To understand and that we need to spend more than one hundred (!) hours.
There is a lot of literature on this subject, so why do you found only " a few short articles and videos" is absolutely unclear.
Start, for example, that with these articles:
D. Berman, A Survey of Deep Learning Methods for Cyber Security
V. Rao Vemuri. Machine learning in computer security
Monowar H. Bhuyan,. Network Anomaly Detection: Methods, Systems and Tools

Delee and you can move on to the more serious sources:
1. Leigh Metcalf, William Casey. Cybersecurity and Applied Mathematics
2. Iván Carrascosa et.al. Data Analytics and Decision Support for Cybersecurity Trends,
Methodologies and Applications.
3. Big Data Analytics in Cybersecurity. Edited by Onur Savas and Julia Deng
4. Brij B. Gupta. Machine Learning for Computer and Cyber Security Principles, Algorithms, and Practices
5. Nour Moustafa, Designing an online and reliable statistical anomaly detection framework for dealing with large high-speed network traffic
6. Joshua Saxe, MALWARE DATA SCIENCE Attack Detection and Attribution
........... etc. - in General, sources mass.
And everything is easily searched online. Good luck
April 4th 20 at 00:50
Can. Start with a subject area - ins here so the side.
What is abnormal activity? What metrics are collected? How to classify/categorize.

Well and further already a trick - having a bunch of signs to make training data and feed it to the ins.
Well, let's say a company has a certain list of computers and servers, employees have a certain level of access, the idea of the project is to create a software which consists of packages neurostock, sharpened on the analysis of the different types of logs, depending on what software the company uses, that is, to specific software separately and their metrics. The neural network must be trained on the logs from the ideal behavior of such a system to record and give the location of any anomalies (for example unregistered device or a sudden elevation of privilege) - Nia_Goyet commented on April 4th 20 at 00:53
@Nia_Goyet, so I do not understand why the second time you question in the comments reprinted. Take highlight metrics, classify, etc.
To train the Ann it is necessary not only in the ideal behavior. - laila commented on April 4th 20 at 00:56
I'm a freshman of faculty of information security, it is difficult to navigate in the subject of artificial intelligence, as I understand it, we first divide the text logs into tokens, and then everything else - Nia_Goyet commented on April 4th 20 at 00:59
@Nia_Goyet, I wrote that you should do FIRST. - laila commented on April 4th 20 at 01:02
April 4th 20 at 00:52
You can, but why if it did almost all of the clouds?
Perhaps I'm not very knowledgeable on the topic of cloud service, but as I understand they need to do this manually, but the aim is to arrange things so that the detection was instantaneous as soon as was noticed anomalous activity, that is, as one huge surveillance system in real time - Nia_Goyet commented on April 4th 20 at 00:55
@Nia_Goyet, there are places where manually, and there are add-ons with artificial intelligence) with regard to instantly and in real time that will never appear in principle by the fact that there are delays in collection of logs, the log analysis, the dynamics and stuff like that. For example, the attempted abuse of power can be easily understood, but the impending DDOS is not very (although AWS WAF
and Shield, as well as other similar products - able). - adeline17 commented on April 4th 20 at 00:58
April 4th 20 at 00:54
identify any abnormal activity
Impossible!
tell us more - Nia_Goyet commented on April 4th 20 at 00:57
@Nia_Goyet, it makes no sense.
@Charlotte72 already told You. - viola_Hodkiewicz78 commented on April 4th 20 at 01:00

Find more questions by tags LoggingArtificial intelligence