ó FRTc@sðddlmZdefd„ƒYZdefd„ƒYZdefd„ƒYZdefd „ƒYZd efd „ƒYZd efd „ƒYZdefd„ƒYZ defd„ƒYZ de fd„ƒYZ de fd„ƒYZ dS(iÿÿÿÿ(tsixtStepcBs)eZdZd„Zd„Zd„ZRS(s! Jobflow Step base class cCs tƒ‚dS(s= :rtype: str :return: URI to the jar N(tNotImplemented(tself((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pytjarscCs tƒ‚dS(sS :rtype: list(str) :return: List of arguments for the step N(R(R((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pytargs%scCs tƒ‚dS(sB :rtype: str :return: The main class name N(R(R((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyt main_class,s(t__name__t __module__t__doc__RRR(((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyRs  tJarStepcBs;eZdZdddd„Zd„Zd„Zd„ZRS(s Custom jar step tTERMINATE_JOB_FLOWcCsO||_||_||_||_t|tjƒrB|g}n||_dS(s  A elastic mapreduce step that executes a jar :type name: str :param name: The name of the step :type jar: str :param jar: S3 URI to the Jar file :type main_class: str :param main_class: The class to execute in the jar :type action_on_failure: str :param action_on_failure: An action, defined in the EMR docs to take on failure. :type step_args: list(str) :param step_args: A list of arguments to pass to the step N(tnamet_jart _main_classtaction_on_failuret isinstanceRt string_typest step_args(RR RRRR((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyt__init__8s     cCs|jS(N(R (R((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyRSscCs&g}|jr"|j|jƒn|S(N(Rtextend(RR((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyRVs cCs|jS(N(R(R((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyR^sN(RRR tNoneRRRR(((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyR 4s   t StreamingStepc BsVeZdZdddddddddd„ Zd„Zd„Zd„Zd„ZRS( s Hadoop streaming step R s3/home/hadoop/contrib/streaming/hadoop-streaming.jarc Cs…||_||_||_||_||_||_||_| |_| |_| |_ t |t j ƒrx|g}n||_ dS(s„ A hadoop streaming elastic mapreduce step :type name: str :param name: The name of the step :type mapper: str :param mapper: The mapper URI :type reducer: str :param reducer: The reducer URI :type combiner: str :param combiner: The combiner URI. Only works for Hadoop 0.20 and later! :type action_on_failure: str :param action_on_failure: An action, defined in the EMR docs to take on failure. :type cache_files: list(str) :param cache_files: A list of cache files to be bundled with the job :type cache_archives: list(str) :param cache_archives: A list of jar archives to be bundled with the job :type step_args: list(str) :param step_args: A list of arguments to pass to the step :type input: str or a list of str :param input: The input uri :type output: str :param output: The output uri :type jar: str :param jar: The hadoop streaming jar. This can be either a local path on the master node, or an s3:// URI. N(R tmappertreducertcombinerRt cache_filestcache_archivestinputtoutputR RRRR( RR RRRRRRRRRR((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyRfs#           cCs|jS(N(R (R((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyR™scCsdS(N(R(R((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyRœscCsyg}|jr"|j|jƒn|jd|jgƒ|jrZ|jd|jgƒn|jr||jd|jgƒn|jddgƒ|jrít|jtƒrÔx=|jD]}|jd|fƒq´Wqí|jd|jfƒn|jr|jd|jfƒn|j rBx'|j D]}|jd|fƒq"Wn|j rux'|j D]}|jd |fƒqUWn|S( Ns-mappers -combiners-reducers-jobconfsmapred.reduce.tasks=0s-inputs-outputs -cacheFiles -cacheArchive( RRRRRRRtlistRRR(RRRt cache_filet cache_archive((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyRŸs.       c CsVd|jj|jj|j|j|j|j|j|j|j |j |j |j f S(Ns‰%s.%s(name=%r, mapper=%r, reducer=%r, action_on_failure=%r, cache_files=%r, cache_archives=%r, step_args=%r, input=%r, output=%r, jar=%r)( t __class__RRR RRRRRRRRR (R((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyt__repr__Äs N( RRR RRRRRR"(((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyRbs  /   %tScriptRunnerStepcBseZdZd„ZRS(sEs3n://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jarcKs#tt|ƒj||j|dS(N(tsuperR#RtScriptRunnerJar(RR tkw((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyRÐs(RRR%R(((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyR#ÌstPigBasecBseZdddgZRS(s4s3n://us-east-1.elasticmapreduce/libs/pig/pig-scripts --base-paths*s3n://us-east-1.elasticmapreduce/libs/pig/(RRtBaseArgs(((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyR'ÔstInstallPigStepcBs eZdZdZdd„ZRS(s! Install pig on emr step s Install PigtlatestcCs\g}|j|jƒ|jdgƒ|jd|gƒtt|ƒj|jd|ƒdS(Ns --install-pigs--pig-versionsR(RR(R$R)RtInstallPigName(Rt pig_versionsR((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyRás (RRR R+R(((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyR)ÚstPigStepcBseZdZdgd„ZRS(s Pig script step R*cCsog}|j|jƒ|jd|gƒ|jddd|gƒ|j|ƒtt|ƒj|d|ƒdS(Ns--pig-versionss--run-pig-scripts--argss-fR(RR(R$R-R(RR tpig_fileR,tpig_argsR((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyRîs  (RRR R(((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyR-éstHiveBasecBseZdddgZRS(s6s3n://us-east-1.elasticmapreduce/libs/hive/hive-scripts --base-paths+s3n://us-east-1.elasticmapreduce/libs/hive/(RRR((((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyR0÷stInstallHiveStepcBs#eZdZdZddd„ZRS(s" Install Hive on EMR step s Install HiveR*cCsg}|j|jƒ|jdgƒ|jd|gƒ|dk r\|jd|gƒntt|ƒj|jd|ƒdS(Ns--install-hives--hive-versionss--hive-site=%sR(RR(RR$R1RtInstallHiveName(Rt hive_versionst hive_siteR((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyRs N(RRR R2RR(((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyR1ýstHiveStepcBseZdZddd„ZRS(s Hive script step R*cCs~g}|j|jƒ|jd|gƒ|jddd|gƒ|dk r^|j|ƒntt|ƒj|d|ƒdS(Ns--hive-versionss--run-hive-scripts--argss-fR(RR(RR$R5R(RR t hive_fileR3t hive_argsR((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyRs N(RRR RR(((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyR5sN( t boto.compatRtobjectRR RR#R'R)R-R0R1R5(((s:/opt/freeware/lib/python2.7/site-packages/boto/emr/step.pyts.j