What is special about cyber defenses? why is it relevant to ML?

huge area, large area that applies of ML, potentially involves human lives
a test bed, what happens if computer had full control
already an area where humans are not involved anymore, thus attacks very relevant
delay in the time when humans find out about flaw

need a well defined system, if non deterministic very difficult, probably dangerous? thus bound danger
how bound the system and make it deterministic. however humans cannot bound
-> need to bound damage caused by an attacker

1. key problem in usage of ml: new forms of attacks for autonomous systems
so far nothing comparable

2. autonomous defense reside on computer. what if that computer is affected.
learning with minimal interaction for both attacker and benign (benign: is agent affected?)
-> minimalist learning
remark: this problem has already been dealt with in statistics
-> new is lack of details? 
-> feature efficient learning, zero shot learning

--> all this becomes very realistic for autonomous systems

how to build a reliable autonomous system on top of 'unreliable' systems (ml, humans)
-> as long as better than before, we won

necessity created by large amounts of data
military: no humans available in context

question: autonomous systems increase productivity or on their own (e.g. replacing human)?

3. attacks on learning to defeat autonomous system. spoil it by prevent it from learning the task correctly
learning system vs. deterministic, logic system, state machine is harder to bring to fail
- complex enough to solve problems?
what about defaulting back to manual?
- convince attacker not to use algorithmic attacks?

setting where machine learning challenges are brought together and need to be solves
-> needed datasets? 
needed data:
currently image data, because it is there and very usable
-> maybe create datasets on our own  
-> data might be there, but is classified. 
-> create synthetic dataset from that 
-> research field on its own


hard problem: detect malicious behavior on  assigned machine, best program learns by itself
-> take human out of the loop

possibly setting up a competition. having a dataset might enable us to set up a competetion
-> problem with overfitting

trace of instructions, thus decides malicious or not. -> very fragile
many companies trying to do this
Can ml actually be successfully employed in this area?

can ml even replace humans?

-> probably a smooth process, in which the amount of humans decreases slowly

problem. semi-autonomous systems, however we do  not trust them due to high false positives, etc.
-> maybe start slowly working of this

more about the way how to get there, e.g. constant improvements on todays semi-autonomous systems

for complete autonomous system:
we are just lacking some (needed) human capabilities
-> start filling the gaps
[agreement that there will be autonomous faculty before autonomous grad students ;-)]

what about autonomous machines on the battlefield
- very vulnerable to attacks. how to defend itself
be careful about the agent. specifying the agent so careful that we can spot deviations
-> human responsible is needed
examples could as well include autonomous sensors or transport, need not be a machine that causes harms
maybe humans are better fit for real physical world
-> chain of reasoning, autonomous systems's reasoning  needs to be transparent 
-> possibility to provide feedback about incorrect behavior?

problem in ai: we are aiming to high.
-> how to break down autonomy to one phd dissertation to solve it
there is already autonomous systems in game playing. what could be the next step?
going either beyond video game, or more sophisticated
need for self-awareness (ability to perceive that something went wrong)
ability risk perception/awareness, ability to judge own behavior
more flexibility, maybe call for human in certain situation. Maybe redundant systems, like in airplane flying
-> ensemble learning
-> problem: this is not for intelligent adversary, but to tackle noise.

actively observing the world - still adversarial examples
egocentric learning or attention for ml and thus autonomous system
break things down to smaller chunks, maybe go for datasets that for example contain audio and vision

a.swami@ieee.org
Anathram Swami