The output of the sigmoid function always ranges between 0 and 1


Problem of dying neuron/dead neuron is addressed



Download 172.44 Kb.
Page2/3
Date11.06.2021
Size172.44 Kb.
#56847
1   2   3
Readings on Activation Function
Problem of dying neuron/dead neuron is addressed by introducing a small slope having the negative values scaled by α enables their corresponding neurons to “stay alive”.

  • The function and its derivative both are monotonic

  • It allows negative value during back propagation

  • It is efficient and easy for computation.

  • Derivative of Leaky is 1 when f(x) > 0 and ranges between 0 and 1 when f(x) < 0.

    • Leaky ReLU does not provide consistent predictions for negative input values.



    PReLu

    • The idea of leaky ReLU can be extended even further.

    • Instead of multiplying x with a constant term we can multiply it with a “

    Download 172.44 Kb.

    Share with your friends:
    1   2   3



    The database is protected by copyright ©ininet.org 2024
    send message
        Main page
    united states
    total number
    vast majority
    federal government
    other side
    next generation
    supreme court
    information contained
    current state
    executive director
    national association
    middle east
    east coast
    general assembly
    private sector
    same time
    west coast
    united kingdom
    united nations
    total amount
    full range
    highest level
    soviet union
    european union
    national academy