class PythonForeachWriter extends ForeachWriter[UnsafeRow]
The class proceeds as follows:
- Rows streamed through a
process()call on the org.apache.spark.sql.execution.streaming.QueryExecutionThread are buffered in theUnsafeRowBuffer. - The WriterThread streams the buffered data to the Python worker. - Once the streaming query ends, close() is called which signals the buffer to mark the end of streaming input. The streaming query execution thread waits for the WriterThread to complete and throws any exceptions seen by the WriterThread.
- Alphabetic
- By Inheritance
- PythonForeachWriter
- ForeachWriter
- Serializable
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new PythonForeachWriter(func: PythonFunction, schema: StructType)
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
close(errorOrNull: Throwable): Unit
Waits for the writer thread to finish evaluating the Python function.
Waits for the writer thread to finish evaluating the Python function. Throws any exceptions seen by the writer thread.
- errorOrNull
the error thrown during processing data or null if there was no error.
- Definition Classes
- PythonForeachWriter → ForeachWriter
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
open(partitionId: Long, version: Long): Boolean
Called when starting to process one partition of new data in the executor.
Called when starting to process one partition of new data in the executor. See the class docs for more information on how to use the
partitionIdandepochId.- partitionId
the partition id.
- returns
trueif the corresponding partition and version id should be processed.falseindicates the partition should be skipped.
- Definition Classes
- PythonForeachWriter → ForeachWriter
-
def
process(value: UnsafeRow): Unit
Called to process the data in the executor side.
Called to process the data in the executor side. This method will be called only if
openreturnstrue.- Definition Classes
- PythonForeachWriter → ForeachWriter
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()